Topic distribution¶

Using the bag of words generated from the corpus, and the topic models generated, I now use them to project publications in the LDA topic space, as well as for the individuals in the School

Setup¶

import pandas as pd
import numpy as np
import cPickle as pkl

import matplotlib.pyplot as plt
%matplotlib inline
rng = np.random.RandomState(1234567)

from gensim import models
from gensim.corpora import Dictionary

lookup_pub = pkl.load(open('../infnet-analysis/data/lookup_pub.pkl', 'rb'))
lookup_poinf = pkl.load(open('../infnet-analysis/data/lookup_poinf.pkl','rb'))
pub_toks = pkl.load(open('../infnet-scrapper/data/pub_toks.pkl','rb'))

from hdbscan import HDBSCAN ## TRYING WITH HDBSCAN:  http://hdbscan.readthedocs.io/en/latest/basic_hdbscan.html

Topic Distribution by Publications¶

# Load Dictionary to convert words to id:
dictionary = pkl.load(open('../topicModel/dictionary.pkl','rb'))

# Convert tokens to bow:
bowified = lambda row: dictionary.doc2bow(row.summary_toks)
pub_toks['bow'] = pub_toks.apply(bowified, axis=1)

pub_toks.head(4)

# load the LDA models:
fullpubLDA = models.LdaModel.load('fullpub.ldamodel')

def inference(ldaModel, ldaVector):
    num_topics = ldaModel.num_topics
    topic_dist = ldaModel[ldaVector]
    
    # index the topic_distribution according to the distribution:
    out = [0]*num_topics
    for (i,v) in topic_dist:
        out[i] = v
    assert len(out) == num_topics
    return out

_inference = lambda row: inference(fullpubLDA, row.bow)
pub_toks['topic_distribution'] = pub_toks.apply(_inference, axis=1)

pub_toks.head(4)

def best_topic(topic_dist):
    """
    Assign the pulbication with the topic that best describes it;
    this is equivalent to the index that have the highest topic_distribution
    """
    a = np.argmax(topic_dist)
    assert a < 20
    assert a >= 0
    return a

pub_toks['best_topic'] = pub_toks.apply(lambda row: best_topic(row.topic_distribution), axis=1)

pub_toks.head(4)

Clustering¶

## Our dataset will be the topic_distribution:
data = pub_toks.topic_distribution.values

Visualisation¶

We can visualise the data in 2D, and color each publication based on the most salient topic it is on:

from sklearn import manifold

`TSNE`¶

Here, use the manifold package from sklearn to reduce the dimensionality of the data for visualisation. For coloring, use the topic that gives the highest probability

x_components = manifold.TSNE(n_components=2, init='pca', random_state=rng).fit_transform(data)

f = plt.figure(figsize=(10,10))
ax = f.add_subplot(111)
ax.scatter(x_components[:,0], x_components[:,1], c=list(pub_toks.best_topic), cmap=plt.cm.jet, s=8)
ax.legend(loc='best')
ax.axis('off')
plt.show()

`MDS`¶

# Multidimensional Scaling
mds_components = manifold.MDS(n_components=2, random_state=rng).fit_transform(data)

f = plt.figure(figsize=(10,10))
ax = f.add_subplot(111)
ax.scatter(mds_components[:,0], mds_components[:,1], c=list(pub_toks.best_topic), cmap=plt.cm.jet, s=8)
ax.legend(loc='best')
ax.axis('off')
plt.show()

While the colors of the publications are based on the topic the publication is salient on (from the LDA), we make no assumption that these are the clusters that are being used.

Hence, now we use some clustering algorithm to cluster our data such that we can color them accordingly. In our clustering algorithms, we have 20 clusters as well, similar to the topics.

KMeans¶

from sklearn.cluster import KMeans

`n_cluster = 20`¶

kmeansClustering = KMeans(n_clusters=20).fit_predict(data)

f = plt.figure(figsize=(20,10))
ax = f.add_subplot(121)
ax.scatter(x_components[:,0], x_components[:,1], c=kmeansClustering, cmap=plt.cm.jet, s=8)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('clustering based on kMeans (using TSNE)')
ax2 = f.add_subplot(122)
ax2.scatter(x_components[:,0], x_components[:,1], c=list(pub_toks.best_topic), cmap=plt.cm.jet, s=8)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('clustering based on best topic')
plt.show()

f = plt.figure(figsize=(20,10))
ax = f.add_subplot(121)
ax.scatter(mds_components[:,0], mds_components[:,1], c=kmeansClustering, cmap=plt.cm.jet, s=20)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('Clustering baed on kMeans (mds)')

ax2 = f.add_subplot(122)
ax2.scatter(mds_components[:,0], mds_components[:,1], c=list(pub_toks.best_topic), cmap=plt.cm.jet, s=20)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('Clustering based on best_topic (mds)')
plt.show()
plt.show()

`n_cluster = 30`¶

kmeansClustering30 = KMeans(n_clusters=30).fit_predict(data)

f = plt.figure(figsize=(20,10))
ax = f.add_subplot(121)
ax.scatter(x_components[:,0], x_components[:,1], c=kmeansClustering30, cmap=plt.cm.jet, s=8)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('clustering based on kMeans (using TSNE)')
ax2 = f.add_subplot(122)
ax2.scatter(x_components[:,0], x_components[:,1], c=list(pub_toks.best_topic), cmap=plt.cm.jet, s=8)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('clustering based on best topic')
plt.show()

`n_cluster = 10`¶

kmeansClustering10 = KMeans(n_clusters=10).fit_predict(data)

f = plt.figure(figsize=(20,10))
ax = f.add_subplot(121)
ax.scatter(x_components[:,0], x_components[:,1], c=kmeansClustering10, cmap=plt.cm.jet, s=8)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('clustering based on kMeans (using TSNE)')
ax2 = f.add_subplot(122)
ax2.scatter(x_components[:,0], x_components[:,1], c=list(pub_toks.best_topic), cmap=plt.cm.jet, s=8)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('clustering based on best topic')
plt.show()

DBScan¶

from sklearn.cluster import DBSCAN

dbscan_pub = DBSCAN().fit(data)

dbscan_clusters = dbscan_pub.labels_
n_clusters_ = len(set(dbscan_clusters)) - (1 if -1 in dbscan_clusters else 0)
print 'number of clusters:', n_clusters_

number of clusters: 1

dbscan_pub = DBSCAN().fit(x_components)

dbscan_clusters = dbscan_pub.labels_
n_clusters_ = len(set(dbscan_clusters)) - (1 if -1 in dbscan_clusters else 0)
print 'number of clusters:', n_clusters_

number of clusters: 203

f = plt.figure(figsize=(20, 10))
ax = f.add_subplot(121)
color_palette = sns.color_palette('Paired', 203)
cluster_colors = [
    color_palette[x] if x >= 0 else (0.5, 0.5, 0.5) for x in dbscan_clusters
]
ax.scatter(
    x_components[:, 0], x_components[:, 1], c=cluster_colors, s=50, alpha=.25)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('clustering based on hdbscan (using TSNE)')
ax2 = f.add_subplot(122)
ax2.scatter(
    x_components[:, 0],
    x_components[:, 1],
    c=list(pub_toks.best_topic),
    cmap=plt.cm.jet,
    s=50,
    alpha=.25)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('clustering based on best topic')
plt.show()

dbscan_pub_mds = DBSCAN().fit(mds_components)

dbscan_pub_mds_clusters = dbscan_pub_mds.labels_
n_clusters_ = len(set(dbscan_pub_mds_clusters)) - (1 if -1 in dbscan_pub_mds_clusters else 0)
print 'number of clusters:', n_clusters_

number of clusters: 1

HDBSCAN¶

from hdbscan import HDBSCAN
import hdbscan

hdbscan_cluster = HDBSCAN().fit(data)

n_clusters_ = len(set(hdbscan_cluster.labels_)) - (1 if -1 in hdbscan_cluster.labels_ else 0)
print 'number of clusters:', n_clusters_

number of clusters: 102

f = plt.figure()
ax = f.add_subplot(111)
ax.hist(hdbscan_cluster.labels_,bins=50);
plt.show()

f = plt.figure(figsize=(20, 10))
ax = f.add_subplot(121)
color_palette = sns.color_palette('Paired', 102)
cluster_colors = [
    color_palette[x] if x >= 0 else (0.5, 0.5, 0.5)
    for x in hdbscan_cluster.labels_
]
#  colors weighted according to te probability of being in the cluster
cluster_member_colors = [
    sns.desaturate(x, p)
    for x, p in zip(cluster_colors, hdbscan_cluster.probabilities_)
]
ax.scatter(
    *x_components.T, s=50, linewidth=0, c=cluster_member_colors, alpha=0.25)

# ax.scatter(x_components[:,0], x_components[:,1], c=hdbscan_cluster.labels_, cmap=plt.cm.jet, s=8)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('clustering based on HDBSCAN (using TSNE)')
ax2 = f.add_subplot(122)
ax2.scatter(
    x_components[:, 0],
    x_components[:, 1],
    c=list(pub_toks.best_topic),
    cmap=plt.cm.jet,
    s=50,
    alpha=.25)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('clustering based on best topic')
plt.show()

Soft clustering in HDBSCAN*¶

clusterer = HDBSCAN(prediction_data=True).fit(data)

n_clusters_ = len(set(hdbscan_cluster.labels_)) - (1 if -1 in hdbscan_cluster.labels_ else 0)
print 'number of clusters:', n_clusters_

number of clusters: 102

f = plt.figure()
ax = f.add_subplot(111)
ax.hist(clusterer.labels_,bins=50);
plt.show()

f = plt.figure(figsize=(20, 20))
ax = f.add_subplot(221)

soft_clusters = hdbscan.all_points_membership_vectors(clusterer)
color_palette = sns.color_palette('Paired', 102)
cluster_colors = [color_palette[np.argmax(x)] for x in soft_clusters]
ax.scatter(*x_components.T, s=50, linewidth=0, c=cluster_colors, alpha=0.25)

# ax.scatter(x_components[:,0], x_components[:,1], c=hdbscan_cluster.labels_, cmap=plt.cm.jet, s=8)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('soft clustering based on HDBSCAN (using TSNE)')
ax2 = f.add_subplot(222)
ax2.scatter(
    x_components[:, 0],
    x_components[:, 1],
    c=list(pub_toks.best_topic),
    cmap=plt.cm.jet,
    s=50,
    alpha=.25)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('clustering based on best topic')

ax3 = f.add_subplot(223)
cluster_colors = [
    sns.desaturate(color_palette[np.argmax(x)], np.max(x))
    for x in soft_clusters
]
ax3.scatter(*x_components.T, s=50, linewidth=0, c=cluster_colors, alpha=0.25)
ax3.axis('off')
ax3.set_title(
    'soft clustering based on HDBSCAN, desaturated based on cluster probability (using TSNE)'
)
plt.show()

Topic Distribution of PoInf¶

# Create a new pandas table that merge the lookup_poinf and lookup_pub
# each pub in lookup_pub have a collab_id that have a list of collaborators by id
# We can ignore those that are not in the list of id for PoInf

# Create the list of ids for easy checking:
poinf_id = set(lookup_poinf.index)

# we can now create such an index:
pub_mapping = {str(_id):set() for _id in list(poinf_id)}

for row in lookup_pub.iterrows():
    pub_id = row[0]
    collab_ids = row[1]['collab_id']
    for _id in collab_ids:
        if _id in poinf_id:
            pub_mapping[_id].add(pub_id)

row_list = [{'id':k, 'pub_ids':v} for (k,v) in pub_mapping.items()]

# Add these pub_ids to the pandas df:
df_pubmapping = pd.DataFrame(row_list)

lookup_poinf_more = lookup_poinf.join(df_pubmapping.set_index('id'))

lookup_poinf_more.iloc[20:24]

def getToks(pub_ids):
    out = []
    try:
        if len(pub_ids):
            for pub_id in pub_ids:
                out.extend(pub_toks[pub_toks.index == pub_id].summary_toks)
            # Convert the list of lists to a single list:
            out = [tok for tokList in out for tok in tokList]
    except TypeError:
        print(pub_ids)
        
    return out

lookup_poinf_more['summary_toks'] = lookup_poinf_more.apply(lambda row: getToks(row.pub_ids), axis=1)

nan
nan
nan
nan

# Conert to BOW using bowified:
lookup_poinf_more['bow'] = lookup_poinf_more.apply(bowified, axis=1)

lookup_poinf_more['topic_distribution'] = lookup_poinf_more.apply(_inference, axis=1)

lookup_poinf_more.head(2)

len(lookup_poinf_more)

296

lookup_poinf_more['remove_drop'] = lookup_poinf_more.apply(lambda row: len(row.bow) == 0, axis=1)

# Remove individuals that does not have any bow:
lookup_poinf_more_drop = lookup_poinf_more.drop(lookup_poinf_more[lookup_poinf_more.remove_drop==True].index)

len(lookup_poinf_more_drop)

219

lookup_poinf_more_drop['best_topic'] = lookup_poinf_more_drop.apply(lambda row: best_topic(row.topic_distribution), axis=1)

/Users/weiting/miniconda3/envs/py27/lib/python2.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  """Entry point for launching an IPython kernel.

lookup_poinf_more_drop.head(2)

Clustering¶

## Our dataset will be the topic_distribution:
data_poinf = lookup_poinf_more_drop.topic_distribution.values

data_poinf = list(data_poinf)

poinf_tsne = manifold.TSNE(n_components=2, init='pca', random_state=rng).fit_transform(data_poinf)

Visualisation¶

`TSNE`¶

f = plt.figure(figsize=(10, 10))
ax = f.add_subplot(111)
ax.scatter(
    poinf_tsne[:, 0],
    poinf_tsne[:, 1],
    c=list(lookup_poinf_more_drop.best_topic),
    cmap=plt.cm.jet,
    s=50,
    alpha=.5)
ax.legend(loc='best')
ax.axis('off')
plt.show()

poinf_tsne.shape

(219, 2)

`MDS`¶

# Multidimensional Scaling
mds_poinf = manifold.MDS(n_components=2, random_state=rng).fit_transform(data_poinf)

f = plt.figure(figsize=(10,10))
ax = f.add_subplot(111)
ax.scatter(mds_poinf[:,0], mds_poinf[:,1], c=list(lookup_poinf_more_drop.best_topic), cmap=plt.cm.jet, s=50, alpha=.5)
ax.legend(loc='best')
ax.axis('off')
plt.show()

K means¶

kmeansClustering = KMeans(n_clusters=20).fit_predict(data_poinf)

f = plt.figure(figsize=(20, 10))
ax = f.add_subplot(121)
ax.scatter(
    poinf_tsne[:, 0],
    poinf_tsne[:, 1],
    c=kmeansClustering,
    cmap=plt.cm.jet,
    s=50,
    alpha=.5)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('clustering based on kMeans (using TSNE)')
ax2 = f.add_subplot(122)
ax2.scatter(
    poinf_tsne[:, 0],
    poinf_tsne[:, 1],
    c=list(lookup_poinf_more_drop.best_topic),
    cmap=plt.cm.jet,
    s=50,
    alpha=.5)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('clustering based on best topic')
plt.show()

f = plt.figure(figsize=(20, 10))
ax = f.add_subplot(121)
ax.scatter(
    mds_poinf[:, 0],
    mds_poinf[:, 1],
    c=kmeansClustering,
    cmap=plt.cm.jet,
    s=50,
    alpha=.5)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('Clustering baed on kMeans (mds)')

ax2 = f.add_subplot(122)
ax2.scatter(
    mds_poinf[:, 0],
    mds_poinf[:, 1],
    c=list(lookup_poinf_more_drop.best_topic),
    cmap=plt.cm.jet,
    s=50,
    alpha=.5)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('Clustering based on best_topic (mds)')
plt.show()
plt.show()

DBScan¶

dbscan = DBSCAN(min_samples=1, algorithm='ball_tree', leaf_size=2).fit(data_poinf)

dbscan_clusters = dbscan.labels_
n_clusters_ = len(set(dbscan_clusters)) - (1 if -1 in dbscan_clusters else 0)
print 'number of clusters:', n_clusters_

dbscan_clusters

array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

# Try with the tsne data:
dbscan_tsne = DBSCAN().fit(x)

dbscan_tsne_clusters = dbscan_tsne.labels_

n_clusters_tsne = len(set(dbscan_tsne_clusters)) - (1 if -1 in dbscan_tsne_clusters else 0)
print 'number of clusters:', n_clusters_

number of clusters: 1

HDBSCAN¶

hdbscan_cluster_poinf = HDBSCAN().fit(data_poinf)

n_clusters_ = len(set(hdbscan_cluster_poinf.labels_)) - (1 if -1 in hdbscan_cluster_poinf.labels_ else 0)
print 'number of clusters:', n_clusters_


f = plt.figure()
ax = f.add_subplot(111)
ax.hist(hdbscan_cluster_poinf.labels_,bins=50);
plt.show()

number of clusters: 2

f = plt.figure(figsize=(20,10))
ax = f.add_subplot(121)
color_palette = sns.color_palette('husl', 103)
cluster_colors = [color_palette[x] if x >= 0
                  else (0.5, 0.5, 0.5)
                  for x in hdbscan_cluster_poinf.labels_]
#  colors weighted according to te probability of being in the cluster
cluster_member_colors = [sns.desaturate(x, p) for x, p in
                         zip(cluster_colors, hdbscan_cluster_poinf.probabilities_)] 
ax.scatter(*poinf_tsne.T, s=50, linewidth=0, c=cluster_member_colors, alpha=0.25)

# ax.scatter(x_components[:,0], x_components[:,1], c=hdbscan_cluster.labels_, cmap=plt.cm.jet, s=8)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('clustering based on HDBSCAN (using TSNE)')

ax2 = f.add_subplot(122)
ax2.scatter(poinf_tsne[:,0], poinf_tsne[:,1], c=list(lookup_poinf_more_drop.best_topic), cmap=plt.cm.jet, s=50, alpha=.5)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('clustering based on best topic')

plt.show()

Soft Clustering of poInf¶

cluster_poinf = HDBSCAN(prediction_data=True).fit(data_poinf)

n_clusters_ = len(set(cluster_poinf.labels_)) - (1 if -1 in cluster_poinf.labels_ else 0)
print 'number of clusters:', n_clusters_

f = plt.figure()
ax = f.add_subplot(111)
ax.hist(cluster_poinf.labels_,bins=50);
plt.show()

number of clusters: 2

f = plt.figure(figsize=(16, 16))
ax = f.add_subplot(221)

soft_clusters = hdbscan.all_points_membership_vectors(cluster_poinf)
color_palette = sns.color_palette('husl', 2)
cluster_colors = [color_palette[np.argmax(x)] for x in soft_clusters]
ax.scatter(*poinf_tsne.T, s=70, linewidth=0, c=cluster_colors, alpha=0.5)
# ax.scatter(x_components[:,0], x_components[:,1], c=hdbscan_cluster.labels_, cmap=plt.cm.jet, s=8)
ax.legend(loc='best')
ax.axis('off')
ax.set_title('soft clustering based on HDBSCAN (using TSNE)')

ax2 = f.add_subplot(222)
ax2.scatter(
    poinf_tsne[:, 0],
    poinf_tsne[:, 1],
    c=list(lookup_poinf_more_drop.best_topic),
    cmap=plt.cm.jet,
    s=70,
    alpha=.5)
ax2.legend(loc='best')
ax2.axis('off')
ax2.set_title('clustering based on best topic')

ax3 = f.add_subplot(223)
cluster_colors = [
    sns.desaturate(color_palette[np.argmax(x)], np.max(x))
    for x in soft_clusters
]
ax3.scatter(*poinf_tsne.T, s=70, linewidth=0, c=cluster_colors, alpha=0.5)
ax3.axis('off')
ax3.set_title(
    'soft clustering based on HDBSCAN, desaturated based on cluster probability (using TSNE)'
)
plt.show()

soft_clusters.shape

(219, 2)

classDef = np.argmax(soft_clusters,axis=1)

class1 = classDef[classDef==0]

class2 = classDef[classDef==1]

lookup_poinf_more_drop.iloc[classDef==1]

lookup_poinf_more_drop.iloc[classDef==0]

	year	summary_toks	bow
pub_id
400818dc-63af-4a26-80c5-906f98e1f8ab	1989	[balloon, stabil, analysi, jet, hmode, dischar...	[(40, 2), (156, 1), (223, 2), (293, 2), (328, ...
18b1a861-afef-4fff-bc80-d02e05be18c4	2013	[queri, process, data, integr, chapter, illust...	[(40, 1), (229, 1), (253, 1), (319, 2), (330, ...
309fdbfc-227b-4588-9264-f0f4e3cadfcb	1994	[comprehens, syntax, syntax, comprehens, close...	[(90, 3), (123, 1), (186, 1), (189, 5), (196, ...
d5814bab-5fc2-4c31-92b7-543c7ce75cb4	2012	[evalu, speaker, verif, secur, detect, hmmbase...	[(18, 1), (28, 1), (29, 2), (30, 1), (63, 1), ...

	year	summary_toks	bow	topic_distribution
pub_id
400818dc-63af-4a26-80c5-906f98e1f8ab	1989	[balloon, stabil, analysi, jet, hmode, dischar...	[(40, 2), (156, 1), (223, 2), (293, 2), (328, ...	[0, 0.0982922003593, 0, 0.0248418858752, 0.422...
18b1a861-afef-4fff-bc80-d02e05be18c4	2013	[queri, process, data, integr, chapter, illust...	[(40, 1), (229, 1), (253, 1), (319, 2), (330, ...	[0, 0, 0, 0, 0, 0.153429075164, 0, 0, 0.341365...
309fdbfc-227b-4588-9264-f0f4e3cadfcb	1994	[comprehens, syntax, syntax, comprehens, close...	[(90, 3), (123, 1), (186, 1), (189, 5), (196, ...	[0, 0.298135265913, 0, 0, 0, 0, 0, 0, 0, 0.181...
d5814bab-5fc2-4c31-92b7-543c7ce75cb4	2012	[evalu, speaker, verif, secur, detect, hmmbase...	[(18, 1), (28, 1), (29, 2), (30, 1), (63, 1), ...	[0, 0, 0, 0, 0.132472201269, 0, 0, 0, 0.122855...

	year	summary_toks	bow	topic_distribution	best_topic
pub_id
400818dc-63af-4a26-80c5-906f98e1f8ab	1989	[balloon, stabil, analysi, jet, hmode, dischar...	[(40, 2), (156, 1), (223, 2), (293, 2), (328, ...	[0, 0.0982922003593, 0, 0.0248418858752, 0.422...	4
18b1a861-afef-4fff-bc80-d02e05be18c4	2013	[queri, process, data, integr, chapter, illust...	[(40, 1), (229, 1), (253, 1), (319, 2), (330, ...	[0, 0, 0, 0, 0, 0.153429075164, 0, 0, 0.341365...	18
309fdbfc-227b-4588-9264-f0f4e3cadfcb	1994	[comprehens, syntax, syntax, comprehens, close...	[(90, 3), (123, 1), (186, 1), (189, 5), (196, ...	[0, 0.298135265913, 0, 0, 0, 0, 0, 0, 0, 0.181...	18
d5814bab-5fc2-4c31-92b7-543c7ce75cb4	2012	[evalu, speaker, verif, secur, detect, hmmbase...	[(18, 1), (28, 1), (29, 2), (30, 1), (63, 1), ...	[0, 0, 0, 0, 0.132472201269, 0, 0, 0, 0.122855...	13

	last_name	first_name	perseonal_url	position	parent	institute	full_name	institute_class	alias	pub_ids	toks
id
0ed800f5-a3a0-47d7-a8b3-f97a4f2b6931	steuwer	michel	http://www.research.ed.ac.uk/portal/en/persons...	unknown	institute for computing systems architecture	laboratory for foundations of computer science	steuwer michel	3	steuwer, m.	{20cb2fdd-6d93-40b9-9cab-e9d818eb166e, b74a3be...	[[highlevel, program, medic, imag, multigpu, s...
102286ee-5f21-4aed-abfd-e4ea1a615223	oberlander	jon	http://www.research.ed.ac.uk/portal/en/persons...	professor	school of informatics	institute of language cognition and computation	oberlander jon	2	oberlander, j.	{5be3a6b1-5ee4-4a39-9fff-88b22238fb98, 4629e88...	[[verbal, effect, visual, program, inform, typ...
10ff8e7a-53b2-4d2f-adad-ef695bc595a7	wen	zhenyu	http://www.research.ed.ac.uk/portal/en/persons...	research associate	school of informatics	laboratory for foundations of computer science	wen zhenyu	3	wen, z.	{872e450f-87e9-4956-9678-5b09f3cd4f84, a4cad99...	[[cost, effici, schedul, mapreduc, applic, pub...
114a78ef-c940-4653-8429-7c2897a96043	jovanovic	jelena	http://www.research.ed.ac.uk/portal/en/persons...	visitor official visitor	school of informatics	institute of language cognition and computation	jovanovic jelena	2	jovanovic, j.	{b4b0b45d-72d9-4f39-929c-ea451288f253}	[[analyticsbas, framework, support, teach, lea...

	last_name	first_name	perseonal_url	position	parent	institute	full_name	institute_class	alias	pub_ids	toks	summary_toks	bow	topic_distribution
id
003ec9bb-18aa-4e6e-95e9-359f0968262a	gray	gavin	http://www.research.ed.ac.uk/portal/en/persons...	research assistant	school of informatics	institute for computing systems architecture	gray gavin	5	NaN	{}	[]	[]	[]	[0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.0...
010f9bf0-c04c-4cfb-ab3d-ca150de1e706	jackson	paul	http://www.research.ed.ac.uk/portal/en/persons...	senior lecturer	school of informatics	institute for computing systems architecture	jackson paul	5	jackson, p. b.\|jackson, p.	{c5754b06-fcf9-4362-aa3a-1142589b5402, 167c4b6...	[nuprl, use, circuit, design, nuprl, interact,...	[nuprl, use, circuit, design, nuprl, interact,...	[(22, 2), (27, 1), (29, 15), (39, 1), (40, 1),...	[0, 0.326083956289, 0.120584281776, 0.05047796...

	last_name	first_name	perseonal_url	position	parent	institute	full_name	institute_class	alias	pub_ids	summary_toks	bow	topic_distribution	remove_drop	best_topic
id
02c86de2-0fc9-4f6d-aee9-93b0f7557c84	franke	bjoern	http://www.research.ed.ac.uk/portal/en/persons...	reader	school of informatics	institute of language cognition and computation	franke bjoern	2	franke, b.	{9a3368cc-e69d-4ecf-bad1-b43ab0ac89a8, ab3fccd...	[use, genet, program, sourcelevel, data, assig...	[(1, 1), (3, 5), (6, 2), (10, 1), (11, 1), (13...	[0.0467349416537, 0, 0.0134058642663, 0.685572...	False	3
0b2fae7b-cf7f-4f8b-a92c-4fa055ff9d63	bhatotia	pramod	http://www.research.ed.ac.uk/portal/en/persons...	senior lecturer in computing systems architecture	school of informatics	institute for computing systems architecture	bhatotia pramod	5	bhatotia, p.	{ea82758c-a623-40b4-9a11-1147103f2949, bedeb6e...	[increment, mapreduc, comput, larg, scale, big...	[(23, 1), (28, 1), (34, 1), (45, 4), (48, 1), ...	[0.0244220477869, 0, 0, 0.398082836141, 0, 0, ...	False	3
0d940898-e2a7-4262-bf06-5b146fb79ba2	spink	tom	http://www.research.ed.ac.uk/portal/en/persons...	research associate	school of informatics	institute of language cognition and computation	spink tom	2	spink, t.	{ab3fccd9-365d-4757-8d2c-f62aaeb59791, e6a0f61...	[hardwar, acceler, crossarchitectur, fullsyste...	[(13, 3), (15, 2), (45, 1), (58, 1), (65, 3), ...	[0.0507503450478, 0, 0, 0.631969091882, 0, 0, ...	False	3
0ed800f5-a3a0-47d7-a8b3-f97a4f2b6931	steuwer	michel	http://www.research.ed.ac.uk/portal/en/persons...	unknown	institute for computing systems architecture	laboratory for foundations of computer science	steuwer michel	3	steuwer, m.	{20cb2fdd-6d93-40b9-9cab-e9d818eb166e, b74a3be...	[highlevel, program, medic, imag, multigpu, sy...	[(5, 31), (6, 1), (11, 1), (15, 2), (28, 9), (...	[0.0121417985093, 0, 0.0332301562306, 0.620844...	False	3
12b8d4c5-226d-430f-8c55-554aa75fcda8	joshi	arpit	http://www.research.ed.ac.uk/portal/en/persons...	research associate	school of informatics	laboratory for foundations of computer science	joshi arpit	3	joshi, a.	{aece86e1-ac7e-4bd9-8065-0e6386906749, fdc2a8d...	[function, transcript, factor, target, discove...	[(14, 1), (45, 2), (55, 3), (58, 2), (65, 1), ...	[0.11343552675, 0, 0, 0.43129692923, 0.0342080...	False	3
2026e8cc-5cb9-4397-b9a9-490ed120e2a5	grot	boris	http://www.research.ed.ac.uk/portal/en/persons...	lecturer in informatics	school of informatics	institute of language cognition and computation	grot boris	2	grot, b.	{90a9d1e9-1fdc-47dd-8dc0-89dfca0a1828, e4706f7...	[scaleout, processor, scale, datacent, mandat,...	[(14, 1), (15, 3), (17, 1), (25, 3), (32, 2), ...	[0.0844472337689, 0, 0, 0.563768078615, 0.0217...	False	3
3d2352c5-8e16-4434-8802-67d74d0a4b36	viglas	stratis	http://www.research.ed.ac.uk/portal/en/persons...	personal chair of data management on new hardware	school of informatics	laboratory for foundations of computer science	viglas stratis	3	viglas, s. d.\|viglas, s.	{b047bfa3-0bb7-4686-82d1-08161610ab3f, 21a6def...	[model, multithread, queri, execut, chip, mult...	[(5, 2), (11, 2), (14, 4), (22, 1), (23, 1), (...	[0.0258041796291, 0, 0.0348903432504, 0.340716...	False	3
412b9b6c-a9d5-47d6-81af-018323057f36	bodin	bruno	http://www.research.ed.ac.uk/portal/en/persons...	research associate	school of informatics	institute for computing systems architecture	bodin bruno	5	bodin, b.	{6b26c412-60a4-49de-b0de-56228e87e068, e45d745...	[live, evalu, cyclostat, dataflow, graph, cycl...	[(15, 1), (24, 2), (28, 4), (29, 1), (40, 1), ...	[0.0738652190376, 0.0912953966609, 0, 0.401687...	False	3
489d4278-a0a6-4e8b-857a-4ee0e800766f	wagstaff	harry	http://www.research.ed.ac.uk/portal/en/persons...	research associate	school of informatics	institute of language cognition and computation	wagstaff harry	2	wagstaff, h.	{ab3fccd9-365d-4757-8d2c-f62aaeb59791, 1f8be5b...	[hardwar, acceler, crossarchitectur, fullsyste...	[(13, 4), (15, 2), (45, 1), (58, 1), (65, 3), ...	[0.0516633484039, 0, 0, 0.656913243918, 0.0201...	False	3
68ca9564-52a9-46ce-92af-0480a43b555d	cole	murray	http://www.research.ed.ac.uk/portal/en/persons...	reader	school of informatics	institute for computing systems architecture	cole murray	5	cole, m.\|cole, m. i.\|cole, m. u. r. r. a. y.	{d760e34b-9961-443b-9d39-477ff1f2f7ef, 273132e...	[structur, approach, model, perform, system, u...	[(3, 1), (5, 39), (6, 1), (11, 5), (15, 1), (2...	[0.0416107383395, 0.0637323834977, 0.040842143...	False	3
6dc0617d-43af-4c3a-88a6-aadf5bd8f57b	smith	aaron	http://www.research.ed.ac.uk/portal/en/persons...	reader in computing systems architecture	school of informatics	institute for computing systems architecture	smith aaron	5	smith, a.	{b2650c3e-425d-414a-9b7b-305d238bd694, ff72dc6...	[machin, learn, approach, map, stream, workloa...	[(13, 1), (14, 10), (17, 2), (23, 1), (28, 13)...	[0.0691003822472, 0, 0, 0.567096757548, 0.0157...	False	3
754637e1-0756-4d21-9afe-732b294e303f	dubach	christophe	http://www.research.ed.ac.uk/portal/en/persons...	reader	school of informatics	institute for computing systems architecture	dubach christophe	5	dubach, c.	{b2650c3e-425d-414a-9b7b-305d238bd694, fa62108...	[machin, learn, approach, map, stream, workloa...	[(3, 2), (11, 4), (13, 3), (14, 1), (15, 2), (...	[0.0535716362631, 0, 0, 0.691537160327, 0.0227...	False	3
c18e1d0a-166d-4615-b7f8-4eb02e964656	leather	hugh	http://www.research.ed.ac.uk/portal/en/persons...	reader	school of informatics	centre for intelligent systems and their appli...	leather hugh	1	leather, h.	{0b8939ab-2a63-456a-b883-29c10cfc122c, b973d73...	[optim, space, explor, fastflow, parallel, ske...	[(5, 14), (11, 1), (14, 1), (16, 1), (23, 1), ...	[0.0863808447963, 0.026313977147, 0.0232772770...	False	3
d15a43ba-c1c3-46bd-af99-55483ee5d119	petoumenos	pavlos	http://www.research.ed.ac.uk/portal/en/persons...	research associate	school of informatics	institute of perception action and behaviour	petoumenos pavlos	7	petoumenos, p.	{66350be6-d95a-424f-84e8-7243be29c53b, f5858ce...	[alea, finegrain, energi, profil, tool, energi...	[(3, 1), (5, 1), (11, 4), (14, 1), (15, 2), (1...	[0.0947015754768, 0, 0.0381563617703, 0.563898...	False	3
d8071e52-0b22-4b1a-b4aa-3b04ba9c75b7	topham	nigel	http://www.research.ed.ac.uk/portal/en/persons...	chair of computer systems	school of informatics	institute of language cognition and computation	topham nigel	2	topham, n. p.\|topham, n.\|topham, n. p. (ed.)	{bef92040-cde7-4f42-9cbf-d62192de2820, a2a8229...	[earli, resolv, instruct, techniqu, disclos, h...	[(3, 3), (6, 1), (10, 1), (11, 3), (13, 3), (1...	[0.0455292929915, 0.0157290767137, 0.028626599...	False	3
e184b211-081f-450b-8931-b0471a0e0c29	sreekar shenoy	govind	http://www.research.ed.ac.uk/portal/en/persons...	unknown	institute for computing systems architecture	institute for adaptive and neural computation	sreekar shenoy govind	4	sreekar shenoy, g.\|s., g.\|shenoy, g. s.	{5bf2d627-928e-4b29-a4d4-9ae189f8384a, 7579406...	[exploit, tempor, local, network, traffic, use...	[(11, 1), (15, 1), (141, 2), (155, 2), (156, 1...	[0, 0, 0.0765566822572, 0.325480398647, 0.0572...	False	3
eb085c28-d4a0-4d51-83e8-881a148e7fff	kumar	rakesh	http://www.research.ed.ac.uk/portal/en/persons...	unknown	institute for computing systems architecture	UNKNOWN	kumar rakesh	0	kumar, r.	{858f08e1-6a3b-45ed-a970-a41be052ac68, ff971c6...	[assist, static, compil, vector, specul, dynam...	[(15, 1), (17, 1), (24, 1), (26, 1), (28, 49),...	[0.103331177705, 0, 0.0252408426641, 0.5689503...	False	3
f53b1ad8-3f19-4e7e-8a3a-f7996de038c0	nagarajan	vijay	http://www.research.ed.ac.uk/portal/en/persons...	reader	school of informatics	centre for intelligent systems and their appli...	nagarajan vijay	1	nagarajan, v.	{a603b6a9-d7bb-42e8-b343-7e307074618e, 464e8bb...	[understand, effect, data, corrupt, applic, be...	[(3, 1), (6, 5), (10, 1), (13, 2), (17, 2), (2...	[0.0285011260705, 0.0124352771291, 0.025752382...	False	3

Topic distribution¶

Setup¶

Topic Distribution by Publications¶

Clustering¶

Visualisation¶

TSNE¶

MDS¶

KMeans¶

n_cluster = 20¶

n_cluster = 30¶

n_cluster = 10¶

DBScan¶

HDBSCAN¶

Soft clustering in HDBSCAN*¶

Topic Distribution of PoInf¶

Clustering¶

Visualisation¶

TSNE¶

MDS¶

K means¶

DBScan¶

HDBSCAN¶

Soft Clustering of poInf¶

`TSNE`¶

`MDS`¶

`n_cluster = 20`¶

`n_cluster = 30`¶

`n_cluster = 10`¶

`TSNE`¶

`MDS`¶